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ABSTRACT * , | 

State and school district testing activities 
involving basic reading competency . requirements are discussed. The 
purposes' of minimiim competency testing in reading include the 
Videntificatiott of students who need remedial assistance, arid the 
[evaluation of secondary school students regarding graduation 
f^requi^ements. Important considerations regarding test content 
: include: its relationship to the assessment purposes, the role of 
task analysis, the priorities of particular skills, the type of test, 
the ways to include teacher judgment, the practical vs. academic 
;:fiav6r of the test,, the methods for presenting \Very difficult 

materials, and the need for a more developed- theory of reading. 
/Several general properties of good systems for setting standards are 
t presented: (1) The school^s responsibility toward the needs of the 
j.stddents and of the society must be recogniaed. t^) Management 
^standards regarding planning, delegation of responsibility, and 
•.irevieW or quality control should be maintained. (3^ Expectations must 
Abe realistic. (U) Standards should be administered with adequate 
student preparation and feedback. (5) Standards , should be applied 
con sistently and should be acceptable to other institutions, so that 
^^T&fedits will be transferred. (6) Standards should be administered 
fairly so that the involved students feel^they; have been well served. 
.^Author /MV) J 
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• This handout susmarlzes sone state and local district activity 
involving competency requirements In' the Basle Skills area, 
since Fred Finch has done an excellent job reviewing this ac- 
tlvlty I von^t spend time discussing the report. 
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• I"' do want to make two related comments on the trends shown In 
the handout. I believe these coionents are consistent wlth^ 
yesterday*s presentations: 

1. Many of the new competency requirements have been 
adopted without adequate planning and without careful 
consideration of consequences. ^ 

2. Some of the legislation and some of the state and 

^ local board rulings are clearly unsound technically 

and probably tmworkable educationally. Many of these 
flaws could easily be worked out in a pilot study. 

# Despite 'iny concerns^ though^ about some of the ways the movement 
is developing, I still feel it has great potential. .Moreover, 
I believe that we can make a great improvement over current 
practice merely by pulling together some of the best of current 
practice. 
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^aper presented at the annual meeting of the Inteimatlonal Reading Association, 
Miami, May 1977. [Part*of Perspectives in Reading Conference — "Legislation 
ai^d Reading Competency—Understanding the Nationwide Trend Toward Performance 

Assessment Beforehand For Graduation." 



FOUR PARIS TO MY PRESENTATION 

• Purpose of Hlnlmum Compet&ncy Measureinent In Reading 

• Content of Minimal Competency Reading Tests 

• General Properties of Good Standards Setting Systems 

• Methods of Setting Standards ^ 

Purpose of Minimum Competency -Measurement In Reading 

One general purpose Is to Identify students who need remedial assl3tance 

to reach a level of reading skill that will facilitate their future development 

/ " 

Discussion of the future goal for which a minimum reading level should be set * 
tends to focus on concepts such as 

• ability to particjipate. in additional education (either still 

at school or post high school level) 4 , , 

• ability to function in society as a consumer, producer, 

citizen, etc. • ; 'i^:^ 

Important to keep In mind the fact that acquiring basic skills clearly 
doesn't guarantee adequate functioning. 

• Many who have highly developed skills are not doing well. 

• Many have been very successful without skills. 
This purpose raises issues such as: 

• Can there be a single minlni^ standard? 
These two purposes rais^ many questions including: 

• What aboiit students who can'^jb meet the minimum? 

• What about students beyond the .minimum? 

Another clear purpose, very evident In state legislation, is that of limiting 

access to thia high school dipipma to those with at least some minimal level 

* 

\ 

/ 



/ 

of reading skill* This raises many additional issue s' Including: 
• How many "remejliati^students can a school afford? 
not just financial but political i^ue 



(Example of xmfortunate political situation: Local CRT project that went Jbad*) 

We were working with a local school district to develop a CRT system ;f6r 

\ ll 

instructional management. ^ The Intention was to provide each teacher at th^ 
early />elementary grades with objectives and associated test^^ questions. 



We had to consider 



\ 



• Which objectives should be included — at which grades? 

- . ' \ 

• Unfortunately, teaching staff reached the conclusion that the 
Superlntenden'"*: planned to cha^ige emphasis to a teacher accouht- 
ability system. Don't know if this was true, but that is what 
teachers thought. 

• So, as a defense, teachers set objectives and standards that 
they were sure they could meet. ^ 

— Entering first grade students were tested to see 
how they stood on end of first grade objectives 

I — -85% had already mastered the first grade objectives 

This kind of active guerilla warfare can easily occur in any setting when an 

assessment system is 'imposed on the schools. 



CONTENT OF MINIMAL COMPETENCY READING TESTS 

*i 

This presentation is addressed primarily to setting standards but some 

genei>al points can be made regarding test content* 

Relationship to Purpose - Content should be. consistent 
with purpose for minimal competency testing. If primary 
interest is in seeing whether students can read materials 
they will e^ncounter outside of school, test should cover 
such material • At the secondary level,- test should be 
^^dijag comprehension* Then if ^students cannot meet the 
school's standard) follow-up evaluation should be carried 
out using tests as only one tool and building on past 
knowledge about student. 

2. Rolet for Task Analysis - Analyses of reading demands of 
school and adult life need to be carried out and studied. 
Actually we have more to go on in the area of reading than 
In mathematics, writing, or other areas.. 

As part of USOE's Targeted Research and Development 

Program in Reading, an extensive national survey of adult 

reading activities was carried out in the years 1970-75. 

This Included door-to-door surveys to find out what adults 

were reading and for what reason. 

• Exercises developed and administered door- 
to-door to national sample 

This study carried out by ETS can be supplemented by APL, 

NAEP, and a number of state and local district projects. 

3. Decision Re Skills Balance - What- should be the place of 
decoding skills, literal comprehension, and higher level 
inference and evaluation? 

Generally literal comprehension: 

Our BSA - 50% literal comprehension 

40% straightforward Inference 
10% evaluation 



In considerlngl skills balance, aim for an "easy" test, » 

A. We< tend to make tests too difficult for 
best measturement* 

B. It will be very difficult to explain a low 
standard. ^ 

C. In most places our interest is in lower part 
of class. 

4. Form of Test - Should objectively scannable tests be used? 

la combination with other assessment and evaluation techniques? 

5. Link to Teacher Judgment - Need to think about ways that test 
based information can be coznblned 'with teacher judgments. 

(Applies' to 4 and 5) 

Need overall decision - What will be the basic approach? Then 
you need alternative approaches when the basic approach is not 
feasible. * 

n 

6* Flavor of Test - Practical, real-world ^^app lied materials only? 
Balance of real world and academic? / 

7. What About Unreadable Materials? - Does Form 1040 belong on a 
minimal competency reading test? 
(Applies 6 and 7) 

• Typical survey reading test based on school material 

• We have some tests now that have "real life" content — 
APL, LA City, Everyday Skills, BSA 

• Probably will reach some kiad of balance in tests 

• Those unreadable materials pose an interesting problem — 
John Geyer mentioned that many materials that are 
essential go off the readability scales. This shows 

up in^^direct testing. 
m I endorse John Geyer 's appeal re research and rewriting. 
It coujii.also help if we could all try to publicize the 
problems these materials cause. 

8. Need for More Developed Theory of Reading? - Does atheor^tical 
equal amoral? 



A question. I carried away^from the AERA meeting on topic — 
:What should minimum competency reading tests measure?. 

)\ 

I have assumed that the tests should contain materials ' 
and questions that were judged appropriate by reading people 
and community people. The sample- of content should be based 
dri evidence regarding what adults read. The kinds' of ques- 
tions related to our knowledge or assumptions, if necessary, 
about -what adults are trying to learn from materials. 
Questions must be technically sound. 

I. have heard a strong call for much more rigorous 
theoretical analyses and for highly systematic test question 
development rules. 

Research — Let's have it 
Mechanistic item development — I react to 
that the way I do to paint-by-numbers 
as an art form. 



DEFINITIONS — GOING BACK TO THE BIBLE"^ 



• Want to- touch on the issue of definitions and to give you some 
background on the development of "standards," "passing scores," 
and "cutting scores •" "^^^ 

• As my reference book, I am using The Bible (King James Version), 
Old Testament., Book of Judges, Chapter 12. (In case some of you 
don't remember that chapter, it deals with the Gileadites and 
Ephraimltes • ) 

— Gileadites were holding the passes (passages) of Jordan, 
^Ephraimltes wanted to go through these passes • [They 

wanted to earn a passing score,] 
— Gileadites had developed a one item test that they had' 

a lot of confidence in. 
— If they tb.ought someone was an Ephralmite 

"Then said they unto him, say now Shib - bo - leth: 
and he said Sib - bo - leth for he could not frame 
to pronounce it right." 
— Those who said Sib-bo- leth and thus did not achieve a 
(Jordan) passing score were slain 

"••• and there fell at that time forty and two 
thousand • " 

— ^My ETS colleague. Bill Angoff , says that the Ephraimites 
plus some Gileadites with speech handicaps and some other 
non-narrative speakers probably had their heads cut off. 
He feels that this was the origin of the^notion of a 
"cutting score" on a test. 



General Properties >>of Good Standards Setting Systems 

1. Ethics/Responsibility . Competency standards for the diploma 

should be developed by a process that gives major attention 

to the needs of students but which recognizes an institution's 

responsibility to societal* needs. 

-7- 8 



\ --There's an appeal for high school diplomas or some 
related credential to serve as evidence of learning. 
Yet we want our schools to open doors not close them* 

—I thought it worth stating as an issue of ethics . . 

Management Standards * Competency standards should be developed 
using good management standards or procedures, e*g*, with ade-" 
quate attention to: 

planning, 

delegation of responsibility, and 

review and quality control. 

• - j> 

\^ . 

Management standards apply to the steps taken in the design 
and implementation of competency standards for the diploma, so 
that necessary decisions are made by qualified people In a manner 
that provides for consistent quality * 

An Important first step in evaluating the adequacy of an 
institution's approach to standard-setting procedure is to deter- 
mine whether ^overall responsibility for this task has been defined * 

The particular individual (s) who holdNthis responsibility can be 

V 

expected to vary depending on the nature\pf the program, department, 
division, or other part of the ins titution^5i^ thin which evaljiations 
take place* Although someone id.th training ^d experience in the 
field of evaluation would be best for the job,^^ose without such 

background are often selected* In such instances^^ the use of 

i 

trained evaluation consultants will be essential* ^ 

Even when overall responsibility for establishiri^ performance 

\ 

standards can be placed in the hands of a skilled evaliiator, many 



Other Individuals should contribute to the standard-setting 
process* For this purpose, it will be useful to seek assist- 
ance not only ^from faculty and students but also from members 
of the larger coamunlty * 

"^The job of designing and developing competency standards 

' ' ^ vv K 

camot be handled adequately if decisions about particular « ' 

Students must- be made under great time pressure. Proyi4iiig 

adequate calendar time, and also adequate staff fpir** the ^develop-* 

ment of> competency standards, will make it possible to plan and 

j ^>.'^' . 

carry out pilot projects to try out\ standards setting systems. 
Procedures can be established for monitoring' the* application of 

standards and for following up on the students afterwards. Most 

\ 

\ ' 

institutions will find it useful to adopt an iterative process 
for developing competeicy standards. The best available sources 
of inf oinnation should ^be used when setting standards for the. 
first ^group of students tested. The experience of conducting 
the initial evaluations, though, should become part of the back- 
ground for future evaluations. 

Realism . Standards must be set at realistic levels. ^ 

Critics of U.S. high schools often criticize the lackVof 
rigor associated with the evaluation of students * learning Ex- 
periences. Little attention has-been paid to the -reverse problem~ 
setting unrealistically high standards. This problem is most 
likely to occur when the individuals participating in" the setting 
or application of standards do not have sufficient direct knowledge 
of the performance of typical students. It is the responsibility 

10 ■ ■ 
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of the Institution to judge what is reasonable to expect of 
students before making decisions regarding competency standards* 

• We had an example yesterday — as I heard Harry Handler, 
the LA City School Board thought all questions should 
be above average in difficulty because they wanted 
students to be above average. [Setting high standards 
doesi.' t Insure meeting them. ] 

• Clearly we do need high standards. We don't want 
minlmums to become maximums. The direction to move In, 
though, has to be that of ^developing standards at more 
than one level. Many possibilities: 

2 level 3 level • multilevel scales 

hoi^oTs honors 
minimum average 



minimum 



A partial sample of a multilevel scale is the language pro- 
ficiency scales developed by the U.S. Foreign Service Institute 
and used for many years with Peace Corps volunteers: 



Eleinefltary Proficiency \ 

SpeaVing-1. Able*to satisfy routine travel needs and minimun courtesy 
requirements. Can ask and answer questions on topics very familiar to him; 
within the scope of his very limited language experience can understand sim- 
ple questions and statements, allowing for slowed speechj repetition or para- 
phrase; speaking vocabulary inadequate to expn»ss anything but the most 
elementary needs; errors in pronunciation and .graninar are frequent, but can 
be under\stood by a native speaker used to dealing with foreigners attempting 
to speak his language; while topics which are "very familiar" a/id elementary 
needs vary considerably from individual to individual, any person at the" 
S-1 level should be able to order a" simple meal, ask for shelter or lodging, 
ask and give simple directions, make purchases, and tell time. 

Reading-1. Able to read some personal and place names, street signs, ' 
office and shop designations, nunbtrs, and 'isolated words and phrases. «Can 
rtcognize-all the letters in the printed version of an alphabetic system 
and high-frequency elements of a $yllab4»7 or a character system. 
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Llwlfd Hofkinq Proficltncv 

^SpHk1n9«>2. Xbit to- satisfy routint socltl dSMnds and limits work 
mqulrMtnts. Can handlt with confldtnct but nbt-wlth facility most social 
situations Including Introductions and ^casual conversations about, current 
events, as well, as work, family, and autbblooraphlcal Infomation; can han- 
dle limited work requiremnts^ needing help In handling any complications 
or d1ff1cu1ti^»;>can get the sist of most conversations on fion'^technical 
subjects (1.e«', topics which require no spHcialized knowledge) and ha't a 
speaking vocabulary sufficient .to\express himself simply with some circum** 
locutions; accent, though often quite faulty.. Unintelligible; can usually 
handle elementary constructions quite accurately but does not have thorough 
or confident control of the grammar. « 

Read1ng-2« Able to read siirplt prose, in a form equivalent to type- 
script or. printing, on subjects within a familiar context* With extensive 
\kt of a dictionary can get the general sen)e of routine business letters^ 
international news items, or articles in tQchnical fields within his co.'iW 
petence. 

Miniroaa Professional Proficiency 

/ $peaking-3. Able to speak the languade^^wlth sufficient structural ac- 
curacy and vocabulary to participate effectively* in most formal and Informal 
.conversations on practical » social « and professional-topics* Can discuss 
particular interests arui special f1e1d$N0f competence with reasonable east; 
comprehension is quite complete for a normal rate of speech; vocabulary js 
broad enough that he rarely has to grope fdfr a word; accent may be obvious- 
ly foreign; control of grannar good; errors never Interfere with understand- 
ing and rarely disturb the native speaker. 

' Reading- J* Able to read standard newspaper -items addressed to the gen- 
eral reader, routine correspondence, reports and technical material in his 
special field. *Can grasp the essentials of articles of the above types 
without using a dictionary; for accurate understanding moderately frequent 
use of a dictionary is required. Has occasional difficulty with unusually 
complex structures^ and low-frequency idioms. 

Full Professional Proficiency 

Speaking-4. Able to use the language fluently and accurately on all 
levels normally pertinent to professional needs. Can understand and parti- 
cipate in any conversation within the range of his experience with a high 
degree of fluency and precision of vocabulary; would rarely be taken for a 
native speaker, but can respond appropriately even in unfamiliar situations; 
errors of pronunciation and gramnar quite rare; can handle informal inter- • 
preting from and into the language. 

Readingr4. Abfe to read all styles and forms of the language pertinent 
to professional needs. With occasional use of a dictionary can read mod- 
erately difficult prose readily in any ar'ea directed to the general reader « 
and all material in his special field including official and professional 
documents and correspondence; can read reasonably legible handwriting with- 
ouX difficulty. v 

Native or Bilingual Proficiency \ 

Speaking-5. Speaking proficiency equivalent to that of an educated 
native speaker. Has complete fluency in the language such that his speech 
on all levels is fully accepted by educated native speakers in all of its 
features* including breadth of vocabulary and idion, colloquial isms* and 
pertinent cultural references. 

Readiny-5. Reading proficiency equivalerit to that of an educated na- 
tive. Can read extremely difficult and abstract prose, as 'well as highly 
colloquial writings and the classic literary forms of the language. With 
varying degrees of difficul%can read all normal kinds of handwritten 
documents. 
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4. Adequacy of Preparation and Feedback. Standards should be 

administered with an adequate student advisory and giiidance 

system' so that high standards are not accompanied by high 

failure rates « 

— ^This is an area that will need attention* 
Students need to know what will be expected 
of them* Need ongoing help if they can't 
meet standard Initially* May need help even 
after leaving school* Could pick up skills 
and meet standard later* 

5* Consistency * Standards should be applied consistently within 

programs, across programs, and over time* 

— ^This means consistent assessment conditions 

- Probably should mean tests with generous 
« V • time limits or no limit . 



- "Secure" test forms 

- Equating of test forms so a score on a 
new form will have the same meaning 

- Need consistent "exception" procedures* 



6* Acceptability /Transferability * Standards should be backed 

o 

fully by the home institution to increase the likelihood that 

the resulting diploma will be acceptable to other institutions. 

— Need to document and explain your procedures* 
Need public relations work here* 

— Possibly active local efforts could prevent 

state imposed systems* Set your own standards 
in pilot .projects. ^ 

er|c . . "^^"ia 



—May need multiple diplomas or special iaforma- 
• tiqn on diplomas to satisfy all the different 
audiences^ ^ 

\ 

Student Satisfaction ^ Standards shoiild be administered so that 
students involved feel they have been well served. 

Summary: Standards should be FAIR, CONSISTENT, and REALISTIC 
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METHODS OF SETTING STANDARDS 



Three factors will contribute decisively to the standards setting pro- 
cedures of any state or school district. 

• The exter&al requirements of other Institutions, agencies 
and the like, e.g., state departments of education, state 
legislatures 

• The basic educational mission and objectives of the school 
/ district or state 

• The personal development goals of individual students 

The approach to be used may vary depending on whether only an overall 
statement regarding reading competency is sought or whether a number of 
separate decisions abput aspects of reading competency will be made. 

Hy discussion assumes that you have a set of reading tasks or questions 
and want to set a minimum standard (and possibly other higher level standards); 
In some cases you may also want to have a set of reading objectives, some jtest 
questions with each objective, and a standard for each objective. You will 
still need an overall standard. 

Traditional or Arbitrary Method 

This method uses some fixed percentage of correct answers on an examina- 
tion, e.g., 65%, as a passing score or standard. It does not take into account 
the nature of the skills or content tested, the student population, or other 
factors. Even though it is the most widely used of any standard setting method, 
I believe that it has almost nothing to recommend it. 

/ 

( 



John Geyer — In New Jersey the Task Force knew standard should be 65% 

♦ * 

before they were sure what should be measured • 

This Is arbitrary, other methods not arbitrary in that they involve data 
collection and systematic judgment done in ways I consider appropriate* 

There is a sense in which the other methods can also be considered arbit- 

rary. Reading competence comes in all degrees, and probably varies on many 

dimensions. Setting a standar 4^ divides a continuous distribution. 

— Small differences at cut point 
— Error of measurement 

Review of Test Questions or Tasks 

The method involves the use of judges who are familiar with the typical 
reading performance of students. The judges have to be able to agree, in 
general terms, on what would constitute minimal reading competency, e.g., how 
they wotild recognize a student who*was minimally competent or who was not ^ 
minimally competent. They then.>need to work with an examination or set of 
tasks that is already available and which they feel can be reasonably used as 
a basis for making decisions about whether or not a student is a minimally 
competent reader. The judges then examine each question or behavioral state- 
ment of a critical task and decide if a minimally competent student could 
answer the question correctly or perform the task adequately. The standard 
is simply the average number of questions or taslcs that each judge has deter- 
mined should be answered correctly or performed at a specified level of compe- 
tence by minimally competent students. ^ / 

'x 

Judges can also be asked to state- the probability diat a minimally com- 
petent person w'ould answer each question correctly or, they could be asked to 



envision a group of minimally competent students and to estimate the portion 
of that group that would answer each question correctly. In any event, the 
Slim of the proportions represents the minimally acceptable score. Where task 
review is involved, judges will need to decide on the weight to be given to 
different aspects of the task, such as the kind and degree of learning re- 
flected in the product or performance. 

Use of Preselected or Criterion Groups 

This method requires the identification of two groups of students: 

— one above minimum competence . 

— the other below minimum competence 

on the particular attribute for which you are setting the standard. *You may 
also want to have a group of students about whom you are doubtful — students 
that would fall in between the above ^nd the below group. You need to divide 
the students 'into the two (or three) groups on some basis pther than the 
examination that you want to use in your operational program.^ You should use 
teachers* judgments, course grades, special testing, observations, and self 
reports. (Use any information that is available or which you can obtain for a 
reasonable expenditure of money or time.) 

It makes sense to devote more time to this activity than you would want 
to devote in an operational program since ttjis division of the students is 
going to lead to the setting of a standard on a more practical procedure that 
can be used on an ongoing basis. After you have your students divided for 
analysis purposes (you don't have them "^divided In separate rooms), you are then 
ready to administer your examination or your assessment procedure to the dif- 
ferent groups and obtain a distribution of scores. Your competency standard. 



then. Is selected to discriminate between the two groups> If your measure 
does not discriminate it between your two groups (or aindng .your three groups) 
you shotild not use it as a basis £or setting standards. 



Classification Errors 

When you are deciding where to set a standard, you have to consider two 
major types of errors: 



• If the standard is set too high, the probability ^ 
Increases that those who should be considered 

\ coiiq)etent will be classified as not competent. 

• If the standard is set too low, the probabd^llty 
Increases that those who should fall will be passed. 

\ 

Clearly the setting of standards must take into account the costs or losses 
assdciated\wlth mlsclasslflcatlons of each type. In every situation, basic 

principles of fairness, realism, and consistency will apply. ' 

> - -1 

An observation on this process — the job of setting appropriate performance 
Standards is as critical as it is difficult. Developers of evaluation procedures 
as part of minimal competency requirements should allow sufficient time in their 
projects for a thoughtful analysis of the possible approaches. More than one 
method of setting a standard should then be tried, and the results compared and 
evaluated. After this evaluation, a decision can be made regarding the method 
or methods that will be used. 
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